Medical imaging apparatus and method of operating same

ABSTRACT

Provided is a method of operating a medical imaging apparatus, comprising: acquiring a first image of a first type corresponding to a first respiratory state of an object; determining motion information of the object with respect to a respiratory state, based on first and second images of a second type respectively corresponding to the first respiratory state and a second respiratory state of the object; and generating a second image of the first type corresponding to the second respiratory state by applying the motion information to the first image of the first type.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.15/169,889, filed on Jun. 1, 2016, in the U.S. Patent and TrademarkOffice, and claims priority from Korean Patent Application No.10-2015-0097768, filed on Jul. 9, 2015, in the Korean IntellectualProperty Office, the disclosures of which are incorporated herein byreference in their entireties.

BACKGROUND 1. Field

The present disclosure relates to medical imaging apparatuses andmethods of operating the same, and more particularly, to medical imagingapparatuses and operation methods for matching a plurality of images.

2. Description of the Related Art

Medical imaging apparatuses are used to acquire images showing aninternal structure of an object. The medical imaging apparatuses arenon-invasive examination devices that capture and process images ofdetails of structures, tissues, flow of fluids, etc., inside a body andprovide the images to a user. A user, e.g., a medical practitioner, mayuse medical images output from the medical imaging apparatuses todiagnose a patient's condition and diseases. Examples of an apparatusfor capturing and processing a medical image may include a magneticresonance imaging (MRI) apparatus, a computed tomography (CT) apparatus,an optical coherence tomography (OCT) apparatus, a single photonemission computed tomography (SPECT) apparatus, a positron emissiontomography (PET) apparatus, an X-ray apparatus, an ultrasound apparatus,etc. A medical image processing apparatus generates a medical image byprocessing captured image data.

Among medical imaging apparatuses, ultrasound diagnosis apparatusestransmit ultrasound signals generated by transducers of a probe to anobject and receive echo signals reflected from the object, therebyobtaining an image of an internal part of the object. In particular,ultrasound diagnosis apparatuses are used for medical purposes includingobserving an internal area of an object, detecting foreign substances,and assessing injuries. Such ultrasound diagnosis apparatuses providehigh stability, display images in real time, and are safe due to therebeing no radiation exposure, compared to X-ray apparatuses. Therefore,an ultrasound diagnosis apparatus is widely used together with othertypes of imaging diagnosis devices.

SUMMARY

Provided are medical imaging apparatuses and methods of operating thesame whereby an accurate internal structure of an object is provided bymatching different types of images.

Provided are medical imaging apparatuses adapted to provide a moreaccurate image than a single image and a matched image of the relatedart by matching images of different modalities in consideration of arespiratory state of an object.

Provided are non-transitory computer-readable recording media havingrecorded thereon a program for executing a method of operating a medicalimaging apparatus on a computer.

Additional aspects will be set forth in part in the description whichfollows and, in part, will be apparent from the description, or may belearned by practice of the presented exemplary embodiments.

According to an aspect of an exemplary embodiment, a method of operatinga medical imaging apparatus includes: acquiring a first image of a firsttype corresponding to a first respiratory state of an object;determining motion information of the object with respect to arespiratory state, based on first and second images of a second typerespectively corresponding to the first respiratory state and a secondrespiratory state of the object; and generating a second image of thefirst type corresponding to the second respiratory state by applying themotion information to the first image of the first type.

The method may further include matching at least one of the first andsecond images of the first type with at least one of the first andsecond images of the second type.

The matching of the at least one of the first and second images of thefirst type with the at least one of the first and second images of thesecond type may include matching at least two images of the first andsecond types corresponding to a same respiratory state.

The method may further include displaying a matched image.

The method may further include: displaying at least one of the first andsecond images of the first type and at least one of the first and secondimages of the second type; and receiving a user input for selecting theat least one of the first and second images of the first type and the atleast one of the first and second images of the second type.

The method may further include matching the selected at least one imageof the first type with the selected at least one image of the secondtype.

The displaying of the at least one of the first and second images of thefirst type and the at least one of the first and second images of thesecond type may include at least one of: displaying the first and secondimages of the first type and the first and second images of the secondtype; and displaying at least two of the first and second images of thefirst type and the first and second images of the second type in such amanner that the at least two images overlap each other.

The displaying of the at least two of the first and second images of thefirst type and the first and second images of the second type in theoverlapping manner may include at least one of: displaying the first andsecond images of the first type in the overlapping manner; anddisplaying the first and second images of the second type in theoverlapping manner.

The determining of the motion information of the object with respect tothe respiratory state may include: acquiring the first and second imagesof the second type; determining at least one parameter for acquiringmotion information indicating a spatial transformation between the firstand second images of the second type; determining a value of the atleast one parameter based on the spatial transformation therebetween;and determining the motion information based on the determined value ofthe at least one parameter.

The spatial transformation may be based on at least one of a position,rotation, and a size of the object.

The determining of the motion information of the object with respect tothe respiratory state may include acquiring position information of theobject and determining the motion information from the first and secondimages of the second type respectively acquired in the first and secondrespiratory states corresponding to the acquired position information.

The first and second images of the first type may be computed tomography(CT) images, and the first and second images of the second type may beultrasound images.

The first and second respiratory states may be inspiratory andexpiratory states of the object, respectively.

According to an aspect of an exemplary embodiment, a medical imagingapparatus includes: an image processor configured to acquire a firstimage of a first type corresponding to a first respiratory state of anobject; and a controller configured to determine motion information ofthe object with respect to a respiratory state based on first and secondimages of a second type respectively corresponding to the firstrespiratory state and a second respiratory state of the object and togenerate a second image of the first type corresponding to the secondrespiratory state by applying the motion information to the first imageof the first type.

The controller may match at least one of the first and second images ofthe first type with at least one of the first and second images of thesecond type.

The controller may match at least two images of the first and secondtypes corresponding to a same respiratory state.

The medical imaging apparatus may further include: a display configuredto display at least one of the first and second images of the first typeand at least one of the first and second images of the second type; anda user interface configured to receive a user input for selecting the atleast one of the first and second images of the first type and the atleast one of the first and second images of the second type.

The display may display at least two of the first and second images ofthe first type and the first and second images of the second type insuch a manner that the at least two images overlap each other.

The image processor may acquire the first and second images of thesecond type, and the controller may determine at least one parameter foracquiring motion information indicating a spatial transformation betweenthe first and second images of the second type, determine a value of theat least one parameter based on the spatial transformation therebetween,and determine the motion information based on the determined value ofthe at least one parameter.

According to an aspect of an exemplary embodiment, a non-transitorycomputer-readable recording medium has recorded thereon a program forperforming a method of operating a medical imaging apparatus, whereinthe method includes: acquiring a first image of a first typecorresponding to a first respiratory state of an object; determiningmotion information of the object with respect to a respiratory state,based on first and second images of a second type respectivelycorresponding to the first respiratory state and a second respiratorystate of the object; and generating a second image of the first typecorresponding to the second respiratory state by applying the motioninformation to the first image of the first type.

According to an aspect of an exemplary embodiment, a method of operatinga medical imaging apparatus includes: acquiring a first image of a firsttype corresponding to a first respiratory state of an object and imagesof a second type respectively corresponding to a plurality ofrespiratory states; acquiring position information of the object;determining motion information of the object with respect to arespiratory state, based on the acquired position information of theobject and the images of the second type; and generating a second imageof the first type corresponding to a second respiratory state byapplying the motion information to the first image of the first type.

The method may further include: matching at least one of the first andsecond images of the first type with at least one of the images of thesecond type; and displaying a matched image.

The determining of the motion information with respect to therespiratory state may include determining the motion information basedon the position information of the object and anatomical structures inthe images of the second type.

According to an aspect of an exemplary embodiment, a medical imagingapparatus includes: an image processor configured to acquire a firstimage of a first type corresponding to a first respiratory state of anobject and images of a second type respectively corresponding to aplurality of respiratory states; and a controller configured to acquireposition information of the object, determine motion information of theobject with respect to a respiratory state, based on the acquiredposition information of the object and the images of the second type,and generate a second image of the first type corresponding to a secondrespiratory state by applying the motion information to the first imageof the first type.

The medical imaging apparatus may further include a display, and thecontroller may match at least one of the first and second images of thefirst type with at least one of the images of the second type. Thedisplay may display a matched image.

BRIEF DESCRIPTION OF THE DRAWINGS

Exemplary embodiments will now be described more fully hereinafter withreference to the accompanying drawings, in which reference numeralsdenote structural elements:

FIG. 1 is a block diagram of a configuration of an ultrasound diagnosisapparatus according to an exemplary embodiment;

FIG. 2 is a block diagram of a configuration of a wireless probeaccording to an exemplary embodiment;

FIG. 3 is a conceptual diagram for explaining an operation of a medicalimaging apparatus for matching a computed tomography (CT) image with anultrasound image, according to an exemplary embodiment;

FIG. 4A is a block diagram of a configuration of a medical imagingapparatus according to an exemplary embodiment;

FIG. 4B is a block diagram of a configuration of a medical imagingapparatus according to another exemplary embodiment;

FIG. 5A is a flowchart of a method of operating a medical imagingapparatus, according to an exemplary embodiment;

FIG. 5B is a flowchart of a method of operating a medical imagingapparatus, according to another exemplary embodiment;

FIG. 5C is a flowchart of a method of operating a medical imagingapparatus, according to another exemplary embodiment;

FIGS. 6A through 6D are diagrams for explaining a first image of a firsttype, corresponding to a first respiratory state of an object, and asecond image of the first type, corresponding to a_second respiratorystate and being acquired from the first image of the first type,according to an exemplary embodiment;

FIG. 7 is a diagram for explaining a matched image corresponding to arespiratory state, according to an exemplary embodiment;

FIGS. 8A and 8B are diagrams for explaining an example where images offirst and second types are displayed on a medical imaging apparatus,according to an exemplary embodiment; and

FIGS. 9A through 9C are diagrams illustrating diagrams illustratingresults of performing registration on images selected by a user,according to an exemplary embodiment.

DETAILED DESCRIPTION

All terms including descriptive or technical terms which are used hereinshould be construed as having meanings that are obvious to one ofordinary skill in the art. However, the terms may have differentmeanings according to the intention of one of ordinary skill in the art,precedent cases, or the appearance of new technologies. Also, some termsmay be arbitrarily selected by the applicant, and in this case, themeaning of the selected terms will be described in detail in thedetailed description of the invention. Thus, the terms used herein haveto be defined based on the meaning of the terms together with thedescription throughout the specification.

Hereinafter, the terms used in the specification will be brieflydescribed, and then the present invention will be described in detail.

The terms used in this specification are those general terms currentlywidely used in the art in consideration of functions regarding thepresent invention, but the terms may vary according to the intention ofthose of ordinary skill in the art, precedents, or new technology in theart. Also, specified terms may be selected by the applicant, and in thiscase, the detailed meaning thereof will be described in the detaileddescription of the invention. Thus, the terms used in the specificationshould be understood not as simple names but based on the meaning of theterms and the overall description of the invention.

When a part “includes” or “comprises” an element, unless there is aparticular description contrary thereto, the part can further includeother elements, not excluding the other elements. Also, the term “unit”in the embodiments of the inventive concept means a software componentor hardware component such as a field-programmable gate array (FPGA) oran application-specific integrated circuit (ASIC), and performs aspecific function. However, the term “unit” is not limited to softwareor hardware. The “unit” may be formed so as to be in an addressablestorage medium, or may be formed so as to operate one or moreprocessors. Thus, for example, the term “unit” may refer to componentssuch as software components, object-oriented software components, classcomponents, and task components, and may include processes, functions,attributes, procedures, subroutines, segments of program code, drivers,firmware, micro codes, circuits, data, a database, data structures,tables, arrays, or variables. A function provided by the components and“units” may be associated with the smaller number of components and“units”, or may be divided into additional components and “units”.

It will be understood that, although the terms “first”, “second”, etc.may be used herein to describe various elements, these elements shouldnot be limited by these terms. These terms are only used to distinguishone element from another element. Thus, a first element discussed belowcould be termed a second element, and similarly, a second element couldbe termed a first element, without departing from the scope of exampleembodiments. As used herein, the term “and/or” includes any and allcombinations of one or more of the associated listed items. Expressionssuch as “at least one of,” when preceding a list of elements, modify theentire list of elements and do not modify the individual elements of thelist.

Throughout the specification, an “image” may refer to multi-dimensionaldata composed of discrete image elements (e.g., pixels in atwo-dimensional (2D) image and voxels in a three-dimensional (3D)image).

Throughout the specification, an “ultrasound image” refers to an imageof an object, which is obtained using ultrasound waves. Ultrasoundimaging apparatuses transmit ultrasound signals generated by transducersof a probe to an object and receive echo signals reflected from theobject, thereby obtaining at least one image of an internal part of theobject. Furthermore, an ultrasound image may take different forms. Forexample, the ultrasound image may be at least one of an amplitude (A)mode image, a brightness (B) mode image, a color (C) mode image, and aDoppler (D) mode image. In addition, the ultrasound image may be a 2D or3D image.

Furthermore, an “object” may be a human, an animal, or a part of a humanor animal. For example, the object may be an organ (e.g., the liver,heart, womb, brain, breast, or abdomen), a blood vessel, or acombination thereof. Also, the object may be a phantom. The phantommeans a material having a density, an effective atomic number, and avolume that are approximately the same as those of an organism.

Throughout the specification, a “user” may be, but is not limited to, amedical expert, for example, a medical doctor, a nurse, a medicallaboratory technologist, or a medical imaging expert, or a technicianwho repairs medical apparatuses.

Reference will now be made in detail to embodiments, examples of whichare illustrated in the accompanying drawings. In this regard, thepresent embodiments may have different forms and should not be construedas being limited to the descriptions set forth herein.

FIG. 1 is a block diagram of a configuration of an ultrasound diagnosisapparatus 100 according to an exemplary embodiment.

Referring to FIG. 1, the ultrasound diagnosis apparatus 100 according tothe present exemplary embodiment may include a probe 20, an ultrasoundtransceiver 115, an image processor 150, a display 160, a communicationmodule 170, a memory 180, an input device 190, and a controller 195,which may be connected to one another via buses 185. The image processor150 may include an image generator 155, a section information detector130, and the display 160.

It will be understood by those of ordinary skill in the art that theultrasound diagnosis apparatus 100 may further include common componentsother than those illustrated in FIG. 1.

In some embodiments, the ultrasound diagnosis apparatus 100 may be acart type apparatus or a portable type apparatus. Examples of portableultrasound diagnosis apparatuses may include, but are not limited to, apicture archiving and communication system (PACS) viewer, a smartphone,a laptop computer, a personal digital assistant (PDA), and a tablet PC.

The probe 20 transmits ultrasound waves to an object 10 in response to adriving signal applied by the ultrasound transceiver 115 and receivesecho signals reflected by the object 10. The probe 20 includes aplurality of transducers, and the plurality of transducers oscillate inresponse to electric signals and generate acoustic energy, that is,ultrasound waves. Furthermore, the probe 20 may be connected to the mainbody of the ultrasound diagnosis apparatus 100 by wire or wirelessly,and according to embodiments, the ultrasound diagnosis apparatus 100 mayinclude a plurality of probes 20.

A transmitter 110 supplies a driving signal to the probe 20. Thetransmitter 110 includes a pulse generator 112, a transmission delayingunit 114, and a pulser 116. The pulse generator 112 generates pulses forforming transmission ultrasound waves based on a predetermined pulserepetition frequency (PRF), and the transmission delaying unit 114delays the pulses by delay times necessary for determining transmissiondirectionality. The pulses which have been delayed correspond to aplurality of piezoelectric vibrators included in the probe 20,respectively. The pulser 116 applies a driving signal (or a drivingpulse) to the probe 20 based on timing corresponding to each of thepulses which have been delayed.

A receiver 120 generates ultrasound data by processing echo signalsreceived from the probe 20. The receiver 120 may include an amplifier122, an analog-to-digital converter (ADC) 124, a reception delaying unit126, and a summing unit 128. The amplifier 122 amplifies echo signals ineach channel, and the ADC 124 performs analog-to-digital conversion withrespect to the amplified echo signals. The reception delaying unit 126delays digital echo signals output by the ADC 1124 by delay timesnecessary for determining reception directionality, and the summing unit128 generates ultrasound data by summing the echo signals processed bythe reception delaying unit 1126.

The image processor 150 generates an ultrasound image by scan-convertingultrasound data generated by the ultrasound transceiver 115.

The ultrasound image may be not only a grayscale ultrasound imageobtained by scanning an object in an amplitude (A) mode, a brightness(B) mode, and a motion (M) mode, but also a Doppler image showing amovement of an object via a Doppler effect. The Doppler image may be ablood flow Doppler image showing flow of blood (also referred to as acolor Doppler image), a tissue Doppler image showing a movement oftissue, or a spectral Doppler image showing a moving speed of an objectas a waveform.

A B mode processor 141 extracts B mode components from ultrasound dataand processes the B mode components. An image generator 155 may generatean ultrasound image indicating signal intensities as brightness based onthe extracted B mode components 141.

Similarly, a Doppler processor 142 may extract Doppler components fromultrasound data, and the image generator 155 may generate a Dopplerimage indicating a movement of an object as colors or waveforms based onthe extracted Doppler components.

According to an embodiment, the image generator 155 may generate athree-dimensional (3D) ultrasound image via volume-rendering withrespect to volume data and may also generate an elasticity image byimaging deformation of the object 10 due to pressure. Furthermore, theimage generator 155 may display various pieces of additional informationin an ultrasound image by using text and graphics. In addition, thegenerated ultrasound image may be stored in the memory 180.

A display 160 displays the generated ultrasound image. The display 160may display not only an ultrasound image, but also various pieces ofinformation processed by the ultrasound diagnosis apparatus 100 on ascreen image via a graphical user interface (GUI). In addition, theultrasound diagnosis apparatus 100 may include two or more displays 160according to embodiments.

The display 160 may include at least one of a liquid crystal display(LCD), a thin film transistor-LCD (TFT-LCD), an organic light-emittingdiode (OLED) display, a flexible display, a 3D display, and anelectrophoretic display.

Furthermore, when the display 160 and the input device 190 form a layerstructure to form a touch screen, the display 160 may be used as aninput device as well as an output device, via which a user inputsinformation via a touch.

The touch screen may be configured to detect a position of a touchinput, a touched area, and pressure of a touch. The touch screen mayalso be configured to detect both a real touch and a proximity touch.

In the present specification, a ‘real touch’ means that a pointeractually touches a screen, and a ‘proximity touch’ means that a pointerdoes not actually touch a screen but approaches the screen while beingseparated from the screen by a predetermined distance. A ‘pointer’ usedherein means a tool for touching a particular portion on or near adisplayed screen. Examples of the pointer may include a stylus pen and abody part such as a finger.

Although not shown, the ultrasound diagnosis apparatus 100 may includevarious sensors that are disposed within or near the touch screen so asto sense a real touch or proximity touch on the touch screen. A tactilesensor is an example of the sensors for sensing a touch on the touchscreen.

The tactile sensor is used to sense a touch of a particular object tothe same or greater degree than the degree to which a human can sensethe touch. The tactile sensor may detect various pieces of informationincluding the roughness of a contact surface, the hardness of an objectto be touched, the temperature of a point to be touched, etc.

A proximity sensor is another example of the sensors for sensing atouch. The proximity sensor refers to a sensor that senses the presenceof an object that is approaching or is located near a predetermineddetection surface by using the force of an electromagnetic field orinfrared light without mechanical contact.

Examples of the proximity sensor include a transmissive photoelectricsensor, a direct reflective photoelectric sensor, a mirror reflectivephotoelectric sensor, a high-frequency oscillation proximity sensor, acapacitive proximity sensor, a magnetic proximity sensor, an infraredproximity sensor, and the like.

The communication module 170 is connected to a network 30 by wire orwirelessly to communicate with an external device or a server. Thecommunication module 170 may exchange data with a hospital server oranother medical apparatus in a hospital, which is connected thereto viaa PACS. Furthermore, the communication module 170 may perform datacommunication according to the digital imaging and communications inmedicine (DICOM) standard.

The communication module 170 may transmit or receive data related todiagnosis of an object, e.g., an ultrasound image, ultrasound data, andDoppler data of the object, via the network 30 and may also transmit orreceive medical images captured by another medical apparatus, e.g., acomputed tomography (CT) apparatus, a magnetic resonance imaging (MRI)apparatus, or an X-ray apparatus. Furthermore, the communication module170 may receive information about a diagnosis history or medicaltreatment schedule of a patient from a server and utilizes the receivedinformation to diagnose the patient. Furthermore, the communicationmodule 170 may perform data communication not only with a server or amedical apparatus in a hospital, but also with a portable terminal of amedical doctor or patient.

The communication module 170 is connected to the network 30 by wire orwirelessly to exchange data with a server 32, a medical apparatus 34, ora portable terminal 36. The communication module 170 may include one ormore components for communication with external devices. For example,the communication module 1300 may include a local area communicationmodule 171, a wired communication module 172, and a mobile communicationmodule 173.

The local area communication module 171 refers to a module for localarea communication within a predetermined distance. Examples of localarea communication techniques according to an embodiment may include,but are not limited to, wireless LAN, Wi-Fi, Bluetooth, ZigBee, Wi-FiDirect (WFD), ultra wideband (UWB), infrared data association (IrDA),Bluetooth low energy (BLE), and near field communication (NFC).

The wired communication module 172 refers to a module for communicationusing electric signals or optical signals. Examples of wiredcommunication techniques according to an embodiment may includecommunication via a twisted pair cable, a coaxial cable, an opticalfiber cable, and an Ethernet cable.

The mobile communication module 173 transmits or receives wirelesssignals to or from at least one selected from a base station, anexternal terminal, and a server on a mobile communication network. Here,the wireless signal may be a voice call signal, a video call signal, ordata in any one of various formats according to transmission andreception of a text/multimedia message.

The memory 180 stores various data processed by the ultrasound diagnosisapparatus 100. For example, the memory 180 may store medical datarelated to diagnosis of an object, such as ultrasound data and anultrasound image that are input or output, and may also store algorithmsor programs which are to be executed in the ultrasound diagnosisapparatus 100.

The memory 180 may be any of various storage media, e.g., a flashmemory, a hard disk drive, EEPROM, etc. Furthermore, the ultrasounddiagnosis apparatus 100 may utilize web storage or a cloud server thatperforms the storage function of the memory 180 online.

The input device 190 generates input data that the user inputs forcontrolling an operation of the ultrasound diagnosis apparatus 100. Theuser input 190 may include hardware components, such as a keypad, amouse, a touch pad, a track ball, and a jog switch. However, embodimentsare not limited thereto, and the input device 1600 may further includeany of various other input units including an electrocardiogram (ECG)measuring module, a respiration measuring module, a voice recognitionsensor, a gesture recognition sensor, a fingerprint recognition sensor,an iris recognition sensor, a depth sensor, a distance sensor, etc.

In particular, the input device 190 may also include a touch screen inwhich a touch pad forms a layer structure with the display 160.

In this case, according to an exemplary embodiment, the ultrasounddiagnosis apparatus 100 may display an ultrasound image in apredetermined mode and a control panel for the ultrasound image on atouch screen. The ultrasound diagnosis apparatus 100 may also sense auser's touch gesture performed on an ultrasound image via a touchscreen.

According to an exemplary embodiment, the ultrasound diagnosis apparatus100 may include some buttons that are frequently used by a user amongbuttons that are included in a control panel of a general ultrasoundapparatus, and provide the remaining buttons in the form of a graphicaluser interface (GUI) via a touch screen.

The controller 195 may control all operations of the ultrasounddiagnosis apparatus 100. In other words, the controller 195 may controloperations among the probe 20, the ultrasound transceiver 100, the imageprocessor 150, the communication module 170, the memory 180, and theinput device 190 shown in FIG. 1.

All or some of the probe 20, the ultrasound transceiver 115, the imageprocessor 150, the communication module 170, the memory 180, the userinput 190, and the controller 195 may be implemented as softwaremodules. However, embodiments of the present invention are not limitedthereto, and some of the components stated above may be implemented ashardware modules. Also, at least one of the ultrasoundtransmission/reception unit 115, the image processor 150, and thecommunication module 170 may be included in the control unit 195;however, the inventive concept is not limited thereto.

FIG. 2 is a block diagram showing a configuration of a wireless probe2000 according to an embodiment. As described above with reference toFIG. 1, the wireless probe 2000 may include a plurality of transducers,and, according to embodiments, may include some or all of the componentsof the ultrasound transceiver 100 shown in FIG. 1.

The wireless probe 2000 according to the embodiment shown in FIG. 2includes a transmitter 2100, a transducer 2200, and a receiver 2300.Since descriptions thereof are given above with reference to FIG. 1,detailed descriptions thereof will be omitted here. In addition,according to embodiments, the wireless probe 2000 may selectivelyinclude a reception delaying unit 2330 and a summing unit 2340.

The wireless probe 2000 may transmit ultrasound signals to the object10, receive echo signals from the object 10, generate ultrasound data,and wirelessly transmit the ultrasound data to the ultrasound diagnosisapparatus 1000 shown in FIG. 1.

The wireless probe 2000 may be a smart device including a transducerarray that is capable of performing an ultrasound scan. In detail, thewireless probe 2000 is a smart device that acquires ultrasound data byscanning an object via the transducer array. Then, the wireless probe2000 may generate an ultrasound image by using the acquired ultrasounddata and/or display the ultrasound image. The wireless probe 2000 mayinclude a display via which a screen including at least one ultrasoundimage and/or a user interface screen for controlling an operation ofscanning an object may be displayed.

While the user is scanning a predetermined body part of a patient thatis an object by using the wireless probe 2000, the wireless probe 2000and the ultrasound diagnosis apparatus 100 may continue to transmit orreceive certain data therebetween via a wireless network. In detail,while the user is scanning a predetermined body part of a patient thatis an object by using the wireless probe 2000, the wireless probe 2000may transmit ultrasound data to the ultrasound diagnosis apparatus 100in real-time via the wireless network. The ultrasound data may beupdated in real-time as an ultrasound scan continues and then betransmitted from the wireless probe 2000 to the ultrasound diagnosisapparatus 100.

FIG. 3 is a conceptual diagram for explaining an operation of a medicalimaging apparatus for matching a computed tomography (CT) image with anultrasound image, according to an exemplary embodiment.

The medical imaging apparatus may match images of first and second typeswith each other. In this case, the images are of different types, thatis, the first type is different from the second type. For example, theimage of the first type may be one of an ultrasound image, an opticalcoherence tomography (OCT) image, a CT image, a magnetic resonance (MR)image, an X-ray image, a single photon emission computed tomography(SPECT) image, a positron emission tomography (PET) image, a C-armimage, a PET-CT image, a PET-MR image, and a fluoroscopy image. It willbe understood by those of ordinary skill in the art that the image ofthe first type is not limited thereto, and may further include othertypes of images.

In the present specification, even when the images of the first andsecond types are referred to as a CT image and an ultrasound image,respectively, the medical imaging apparatus may match not only the CTand ultrasound images but also the other above-described types of imageswith each other.

FIG. 3 is a schematic diagram for explaining a flow of operationsperformed by a medical imaging apparatus for matching CT and ultrasoundimages.

Referring to 310 of FIG. 3, the medical imaging apparatus may acquire afirst image of a first type and a first image of a second type. Forexample, the medical imaging apparatus may acquire an ultrasound image311 and a CT image 312 that are captured images of the same object. Themedical imaging apparatus may acquire the ultrasound image 311 and theCT image 312 by directly photographing an object or receive them from anexternal device.

In general, a CT image and an ultrasound image are captured when anobject is in a (deep) inspiratory state and an expiratory state,respectively. In this case, anatomical features of the object and/or aposition of a structure in the object may change according to arespiratory state of the object. Thus, anatomical features of the objectin a first image of a first type may be different from those in a secondimage of a second type, or a position of a structure in the first imagemay not coincide with that in the second image. In this case, if themedical imaging apparatus matches the first image of the first type withthe second image of the second type, an error may occur with respect tothe anatomical features of the object or the position of the structurein the first and second images. Thus, whether the anatomical features ofthe object and/or a position of a structure in different types of imagescoincide with each other may be a criterion for matching the differenttypes of images. Furthermore, a respiratory state of the object in whichdifferent types of images are captured may be a criterion for matchingthe images.

Referring to 320 of FIG. 3, the medical imaging apparatus may generatean image 321) of a first type to be matched with the first image of thesecond type. For example, in detail, since anatomical features of theobject and/or a position of a structure may vary according to whetherthe object is in an inspiratory or expiratory state, the medical imagingapparatus may match images captured when the object is in the samerespiratory state. The medical imaging apparatus may generate an imageof the first type corresponding to the same respiratory state as thatdepicted in the first image of the second type.

To generate a second image of the first type to be matched with a secondimage of the second type, the medical imaging apparatus may determinemotion information between the first and second images of the secondtype and apply the motion information to the first image of the firsttype.

Referring to 330 of FIG. 3, the medical imaging apparatus may matchdifferent types of images. The medical imaging apparatus may match a setof the first images of the first and second types and a set of thesecond images of the first and second types, respectively. In this case,each set of images being matched may be captured in the same respiratorystate.

For example, the medical imaging apparatus may match a set of anultrasound image 331 and a CT image 332 corresponding to an expiratorystate and a set of an ultrasound image 333 and a CT image 334corresponding to an inspiratory state, respectively.

Referring to 340 of FIG. 3, the medical imaging apparatus may display amatched image. Furthermore, the medical imaging apparatus may display aplurality of images before they are each matched, together with thematched image.

FIG. 4A is a block diagram of a configuration of a medical imagingapparatus 400 according to an exemplary embodiment.

The medical imaging apparatus according to the present exemplaryembodiment may include an image processor 410 and a controller 420.However, all the components shown in FIG. 4A are not essentialcomponents. The medical imaging apparatus 400 may include more or fewercomponents than those shown in FIG. 4A.

The image processor 410 may acquire a first image of a first typecaptured in a first respiratory state of an object. In detail, the imageprocessor 410 may acquire an image of the object by photographing orscanning the object or receive an image of the object from an externaldevice.

In this case, the external device may be physically independent of themedical imaging apparatus 400. The external device is a device foracquiring, storing, processing, or using data related to an image of anobject, and may be a medical server, a portable terminal, or any othercomputing device for using and processing a medical image. For example,the external device may be a medical diagnosis apparatus included in amedical institution such as a hospital. Furthermore, the external devicemay be a server in a hospital for recording and storing a patient'sclinical history, the medical imaging apparatus 400 used by a medicaldoctor in a hospital to read a medical image, or the like.

Furthermore, the image processor 410 may acquire a first image of asecond type captured in the first respiratory state of the object and asecond image of the second type captured in a second respiratory statethereof. Like in the case of images of the first type, to obtain imagesof the second type, the image processor 410 may acquire an image of theobject by photographing or scanning the object, or receive an image ofthe object from the external device. In this case, the images of thesecond type are different from those of the first type.

Furthermore, the image processor 410 may acquire images of the secondtype captured in a plurality of respiratory states of the object. Theplurality of respiratory states may include first through thirdrespiratory states, and are not limited thereto. A respiratory state maybe classified into a plurality of respiratory states depending on thedegree of breathing of the object.

According to an exemplary embodiment, images of the first and secondtypes are a CT image and an ultrasound image, respectively. In addition,one of ordinary skill in the art will understand that the images of thefirst and second types may be any different types of images and are notlimited to the CT image and the ultrasound image, respectively.

Furthermore, the first and second respiratory states may be inspiratoryand expiratory states of the object, respectively. In this case, arespiratory state is merely classified into the first and secondrespiratory states according to the degree of breathing, but this ismerely an example. The respiratory state may be classified into aplurality of respiratory states according to another criterion.

The controller 420 may determine motion information of the object withrespect to_a respiratory state, based on the first and second images ofthe second type respectively corresponding to the first and secondrespiratory states of the object.

In detail, the controller 420 may determine at least one parameter forobtaining motion information indicating a spatial transformation betweenthe first and second images of the second type. The controller 420 maydetermine at least one parameter that minimizes a spatial transformationbetween the first and second images of the second type. In this case,the spatial transformation may be based on at least one of a position,rotation, and a size of the object. The controller 420 may determinemotion information based on the determined at least one parameter.

To determine the spatial transformation between the first and secondimages of the second type, the controller 420 may calculate a differencebetween the first and second images. The difference between the firstand second images of the second type may be calculated using Equation(1):

Sum((l(p)−l(T*p))̂2)/N   (1)

where T, P, and N respectively denote a spatial transformation, aspatial position, and the number of pixels. The difference between thefirst and second images may be calculated using a difference betweenpixel values as well as various image properties that may be extractedfrom an image, such as an image texture, a histogram, etc.

For example, the controller 420 may determine a value of a parameterthat minimizes a spatial transformation between the first and secondimages of the second type. In this case, one of ordinary skill in theart will understand not only that a value of a parameter may bedetermined if a spatial transformation is minimized but also that if thespatial transformation satisfies a predetermined condition, a value of aparameter may be determined according to the predetermined condition.

The controller 420 may also update a parameter describing a spatialtransformation so as to reduce a difference between the first and secondimages. In this case, the controller 420 may repeat a process ofcalculating the difference between the first and second images andupdating the parameter.

Furthermore, properties of an anatomical structure extracted from eachof the first and second images of the second type may be used indetermining the spatial transformation between the first and secondimages of the second type. The controller 420 may determine the spatialtransformation therebetween by using properties of an anatomicalstructure extracted from each of the first and second images. In detail,at least one of a pixel value difference, an image texture, and ahistogram may be used in extracting properties of an anatomicalstructure and/or determining the spatial transformation between thefirst and second images, and exemplary embodiments are not limitedthereto. In other words, various factors that may be extracted from eachof the first and second images may be used in determining the spatialtransformation therebetween.

Furthermore, the controller 420 may acquire position information of theobject and determine motion information based on the first and secondimages of the second type respectively corresponding to first and secondrespiratory states corresponding to the acquired position information.In this case, the position information may be acquired from a sensor.The sensor may be built into or physically separate from the medicalimaging apparatus 400. The position information may be acquired from thesensor as well as another external device.

The controller 420 may determine motion information of the object withrespect to a respiratory state, based on position information of theobject and images of the second type respectively corresponding to aplurality of respiratory states. The controller 420 may generate asecond image of a first type corresponding to the second respiratorystate by applying the motion information to the first image of the firsttype. Furthermore, the controller 420 may determine motion informationby using anatomical structures included in images of the second type.

For example, the controller 420 may determine the degree oftransformation for a respiratory state by using position information ofthe object and ultrasound images thereof. The position information ofthe object may be acquired from a sensor and may be detected byacquiring position information and ultrasound images with respect totime while free breathing is being performed for a specific time. Inthis case, examples of the sensor may include an electronic sensor, anoptical sensor, etc. for measuring a position and respiration of theobject, and types of the sensor are not limited thereto. Furthermore,the sensor may be attached to an ultrasound probe and measure a positionand respiration of the object. In detail, the controller 420 may acquiretimes when specific directions detected by the sensor (text: specificdirections of the sensor?) correspond to inspiratory and expiratorystates, respectively, and determine motion vectors from ultrasoundimages corresponding to the acquired times.

The controller 420 may determine the degree of transformation in arespiratory state by using ultrasound images. The controller 420 mayacquire ultrasound images with respect to time while free breathing isbeing performed for a specific time and extract anatomical structures orimage properties from the acquired ultrasound images, therebydetermining a spatial transformation between the ultrasound images. Thecontroller 420 may determine motion information (e.g., motion vectors)based on the degree of transformation determined for the respiratorystate.

The controller 420 may generate a second image of the first typecorresponding to the second respiratory state by applying the motioninformation to the first image of the first type.

In addition, the controller 420 may match at least one of the first andsecond images of the first type with at least one of the first andsecond images of the second type. The controller 420 may select imagesof the first and second types according to a predetermined algorithm orcriterion for registration. Furthermore, as described below withreference to FIG. 4B, the controller 420 may match images of the firstand second types with each other based on a user input.

When images corresponding to the same respiratory state overlap eachother, positions of the object in the images may accurately coincidewith each other, compared to when images corresponding to differentrespiratory states overlap. Thus, the controller 420 may match images ofthe first and second types corresponding to the same respiratory statetogether.

The medical imaging apparatus 400 may include a central arithmeticprocessor that controls overall operations of the image processor 410and the controller 420. The central arithmetic processor may beimplemented as an array of a plurality of logic gates or a combinationof a general purpose microprocessor and a program that can be run on thegeneral purpose microprocessor. Furthermore, it will be appreciated bythose of ordinary skill in the art to which the present embodimentpertains that the central arithmetic processor may be formed bydifferent types of hardware.

FIG. 4B is a block diagram of a configuration of a medical imagingapparatus 400 according to another exemplary embodiment.

Referring to FIG. 4B, unlike the medical imaging apparatus 400 of FIG.4A, the medical imaging apparatus 400 according to the present exemplaryembodiment may further include a display 430 and a user interface 440.

Since an image processor 410 and a controller 420 of the medical imagingapparatus 400 of FIG. 4B respectively correspond to the image processor410 and the controller 420 of the medical imaging apparatus 400 of FIG.4A, the same descriptions as already provided with respect to FIG. 4Awill be omitted below.

The display 430 may display at least one of first and second images of afirst type and at least one of first and second images of a second type.

The display 430 may display at least two of the first and second imagesof the first type and the first and second images of the second type insuch a manner that they overlap each other. For example, the display 430may display the first images of the first and second types in such amanner that they overlap each other.

The controller 420 controls the display 430 to display a predeterminedscreen. The display 430 may display the predetermined screen so that auser or patient may visually recognize a predetermined image orinformation. The display 430 may correspond to the display 160 shown inFIG. 1 or be separate from the ultrasound diagnosis apparatus 100 ofFIG. 1.

The display 430 may display a predetermined screen. In detail, thedisplay 430 may display the predetermined screen according to control bythe controller 420. The display 430 includes a display panel (not shown)and displays a user interface screen, a medical image screen, etc. onthe display panel.

The user interface 440 may receive a user input for selecting differenttypes of images to be matched. In detail, the user interface 440 mayreceive a first user input for selecting at least one from among firstand second images of a first type and a second user input for selectingat least one from among first and second images of a second type.

The user interface 440 refers to a device via which data for controllingthe medical imaging apparatus 400 is received from a user. The userinterface 440 may include hardware components, such as a keypad, amouse, a touch panel, a touch screen, a track ball, and a jog switch,but is not limited thereto. The user interface 440 may further includeany of various other input units including an electrocardiogram (ECG)measuring module, a respiration measuring module, a voice recognitionsensor, a gesture recognition sensor, a fingerprint recognition sensor,an iris recognition sensor, a depth sensor, a distance sensor, etc.

The user interface 440 may generate and output a user interface screenfor receiving a predetermined command or data from the user. The userinterface 440 may also receive the predetermined command or data fromthe user via the user interface screen. The user may view the userinterface screen displayed via the display 430 to visually recognizepredetermined information and input a predetermined command or data viathe user interface 440.

For example, the user interface 440 may be formed as a touch pad. Indetail, the user interface 440 includes a touch pad (not shown) combinedwith the display panel in the display 430. In this case, a userinterface screen is output to the display panel. When a predeterminedcommand is input via the user interface screen, the touch pad may detectinformation about the predetermined command and then transmit thedetected information to the controller 420. Then, the controller 420 mayinterpret the detected information to recognize and execute thepredetermined command input by the user.

The medical imaging apparatus 400 may further include a storage unit(not shown) and a communication module (not shown). The storage unit maystore data related to an object (e.g., an ultrasound image, ultrasounddata, scan-related data, data related to diagnosis of a patient, etc.),data transmitted from an external device to the medical imagingapparatus 400, etc. The data transmitted from the external device mayinclude patient-related information, data necessary for diagnosis andtreatment of a patient, a patient's past medical history, a medical worklist corresponding to instructions regarding diagnosis of a patient, andthe like.

The communication module may receive and/or transmit data from and/or toan external device. For example, the communication module may connect toa wireless probe or an external device via a communication network basedon Wi-Fi or Wi-Fi Direct (WFD) technology. In detail, examples of awireless communication network to which the communication module canconnect may include, but are not limited to, Wireless LAN (WLAN), Wi-Fi,Bluetooth, ZigBee, WFD, Ultra Wideband (UWB), Infrared Data Association(IrDA), Bluetooth Low Energy (BLE), and Near Field Communication (NFC).

In this case, the ultrasound imaging apparatus 400 may obtain aplurality of types of images respectively corresponding to respiratorystates by directly photographing or scanning an object or receiving themfrom an external device. The external device may be a storage device.The storage device may be any of various storage media such as a harddisk drive (HDD), Read Only Memory (ROM), Random Access Memory (RAM), aflash memory, and a memory card.

Hereinafter, various operations performed by the medical imagingapparatus 400 and applications thereof will be described in detail.Although none of the image processor 410, the controller 420, and thedisplay 430 are specified, features and aspects that would be clearlyunderstood by and are obvious to those of ordinary skill in the art maybe considered as a typical implementation. The scope of the presentinventive concept is not limited by a name of a particular component orphysical/logical structure.

FIG. 5A is a flowchart of a method of operating the medical imagingapparatus 400, according to an exemplary embodiment.

Referring to FIG. 5A, the medical imaging apparatus 400 may acquire afirst image of a first type corresponding to a first respiratory state(S510). For example, the medical imaging apparatus 400 may acquire afirst CT image captured when an object is in an inspiratory state. Ingeneral, a CT image is captured in a state in which air has been inhaledfor examination of a lesion. Thus, the medical imaging apparatus 400 maygenerate a CT image corresponding to an inspiratory state by usingmotion vectors of different types of images than the CT image. When themedical imaging apparatus 400 is a CT apparatus, the medical imagingapparatus 400 may acquire a CT image by directly photographing anobject. Furthermore, the medical imaging apparatus 400 may acquire a CTimage from an external device.

The medical imaging apparatus 400 may determine motion information ofthe object with respect to a respiratory state, based on first andsecond images of a second type respectively corresponding to the firstrespiratory state and a second respiratory state of the object (S520).

For example, the medical imaging apparatus 400 may acquire first andsecond ultrasound images respectively corresponding to inspiratory andexpiratory states. The medical imaging apparatus 400 may determine atleast one parameter for obtaining motion information indicating aspatial transformation between the first and second ultrasound images,and determine a value of the at least one parameter based on the spatialtransformation. The medical imaging apparatus 400 may determine motioninformation (e.g., a motion vector) based on the determined value of theat least one parameter).

As another example, the medical imaging apparatus 400 may acquireposition information of the object and determine motion informationbased on the first and second ultrasound images respectively acquired inthe inspiratory and expiratory states corresponding to the acquiredposition information.

Furthermore, the medical imaging apparatus 400 may acquire positioninformation of the object and determine motion information of the objectwith respect to a respiratory state based on the position informationand images of the second type respectively corresponding to a pluralityof respiratory states. The motion information may be determined based onthe position information of the object and anatomical structures in theimages of the second type.

The medical imaging apparatus 400 may generate a second image of thefirst type corresponding to the second respiratory state (S530). Indetail, the medical imaging apparatus 400 may generate the second imageof the first type corresponding to the secondary respiratory state byapplying the motion information to the first image of the first type.

For example, the medical imaging apparatus 400 may acquire motioninformation indicating a structural or spatial transformation of theobject from first and second ultrasound images respectivelycorresponding to inspiratory and expiratory states. The medical imagingapparatus 400 may generate a second CT image corresponding to theexpiratory state by applying the motion information to a first CT imagecaptured in the inspiratory state.

In addition, a non-transitory computer-readable recording medium havingrecorded thereon a program for performing a method of operating themedical imaging apparatus 400 may include codes representing the method.In detail, the codes may include a code for acquiring a first image of afirst type corresponding to a first respiratory state of an object, acode for determining motion information of the object with respect to arespiratory state based on first and second images of a second type, anda code for generating a second image of the first type corresponding toa secondary respiratory state by applying the motion information to thefirst image of the first type, but the codes are not limited thereto.

FIG. 5B is a flowchart of a method of operating the medical imagingapparatus 400, according to another exemplary embodiment.

Referring to FIG. 5B, the medical imaging apparatus 400 may match imagesof first and second types together (S540). The medical imaging apparatus400 may match at least one of first and second images of the first typeand at least one of first and second images of the second type. Indetail, the medical imaging apparatus 400 may match the second image ofthe first type with the first image of the second type.

For example, the medical imaging apparatus 400 may match a second CTimage corresponding to an expiratory state with a second ultrasoundimage corresponding to the expiratory state. Furthermore, the medicalimaging apparatus 400 may match a first CT image corresponding to aninspiratory state with a first ultrasound image corresponding to theinspiratory state. In addition, the medical imaging apparatus 400 maymatch different types of images respectively corresponding to differentrespiratory states.

Furthermore, the medical imaging apparatus 400 may match at least one offirst and second images of the first type with at least one of images ofthe second type.

The medical imaging apparatus 400 may display a matched image (S550).Furthermore, the medical imaging apparatus 400 may display the matchedimage together with images before they are matched.

In addition, a non-transitory computer-readable recording medium havingrecorded thereon a program for performing a method of operating themedical imaging apparatus 400 may include codes representing the method.In detail, the codes may include a code for matching images of first andsecond types together and a code for displaying a matched image, but thecodes are not limited thereto.

FIG. 5C is a flowchart of a method of operating the medical imagingapparatus 400, according to another exemplary embodiment. FIG. 5C is aflowchart for explaining a process of receiving a user input forselecting images to be matched from among different types of images.

Referring to FIG. 5C, the medical imaging apparatus 400 may displayimages of a first type and images of a second type (S532). The medicalimaging apparatus 400 may display some of a plurality of images of thefirst type and/or some of a plurality of images of the second type.

The medical imaging apparatus 400 may display at least one of aplurality of images of the first type and at least one of a plurality ofimages of the second type in such a manner that they overlap each other.

Furthermore, the medical imaging apparatus 400 may display at least twoof a plurality of images of the first type in the overlapping mannerwhile displaying at least two of a plurality of images of the secondtype in the overlapping manner.

The medical imaging apparatus 400 may receive a user input for selectingat least one image from among the images of the first type and at leastone image from among the images of the second type (S534).

The medical imaging apparatus 400 may receive a user input for selectinga second ultrasound image and a second CT image, both of whichcorrespond to an expiratory state. The medical imaging apparatus 400 maymatch the second ultrasound image corresponding to the expiratory statewith the second CT image corresponding to the expiratory state, based onthe user input.

In addition, a non-transitory computer-readable recording medium havingrecorded thereon a program for performing a method of operating themedical imaging apparatus 400 may include codes representing the method.In detail, the codes may include a code for matching images of first andsecond types together and a code for receiving a user input forselecting at least one image from among images of the first type and atleast one image from among images of the second type, but the codes arenot limited thereto.

FIGS. 6A through 6D are diagrams for explaining a first image of a firsttype, corresponding to a first respiratory state of an object, and asecond image of the first type, corresponding to a second respiratorystate and being acquired from the first image of the first type,according to an exemplary embodiment. The medical imaging apparatus 400may provide a cross-sectional image of an object and display an internalstructure (e.g., the heart, liver, stomach, etc.) withoutsuperimposition of adjacent structures.

Referring to FIG. 6A, the medical imaging apparatus 400 may acquire afirst CT image 610 captured in an inspiratory state and showing a liver621 and a stomach 624 of a human. A CT image may be generally capturedin an inspiratory state. The medical imaging apparatus 400 may generatea second CT image 620 corresponding to an expiratory state by usingdifferent types of images than a CT image. In this case, the differenttypes of images may be first and second ultrasound images respectivelycorresponding to inspiratory and expiratory states. It will be obviousto those of ordinary skill in the art that the different types of imagesare not limited to ultrasound images. As shown in FIG. 6A, structural orspatial positions of the liver 621, an inferior vena cava 622, an aorta623, the stomach 624, etc. may vary according to a respiratory state.

Referring to FIG. 6B, the medical imaging apparatus 400 may acquire afirst CT image 630 captured in an inspiratory state and showing a heart631, a stomach 641, and a kidney 642 of the human. The medical imagingapparatus 400 may generate a second CT image 640 corresponding to theinspiratory state by using different types of images than a CT image. Asshown in FIG. 6B, structural or spatial positions of the heart 631, thestomach 641, and the kidney 642 may vary according to a respiratorystate.

Referring to FIG. 6C, the medical imaging apparatus 400 may acquire afirst CT image 650 captured in an inspiratory state and showing a liver651, a spine 652, and a kidney 653 of the human. The medical imagingapparatus 400 may generate a second CT image 660 corresponding to anexpiratory state by using different types of images than a CT image. Asshown in FIG. 6C, structural or spatial positions of the liver 651, thespine 652, and the kidney 653 may vary according to a respiratory state.

Referring to FIG. 6D, the medical imaging apparatus 400 may acquire afirst CT image 670 captured in an inspiratory state and showing a liver681, a stomach 682, and an aorta 683 of the human. The medical imagingapparatus 400 may generate a second CT image 680 corresponding to anexpiratory state by using different types of images than a CT image. Asshown in FIG. 6D, structural or spatial positions of the liver 681, thestomach 682, and the aorta 683 may vary according to a respiratorystate.

FIG. 7 is a diagram for explaining a matched image corresponding to arespiratory state, according to an exemplary embodiment.

Referring to 710 of FIG. 7, an ultrasound image is acquired by scanningan object in an expiratory state. As shown in 710 of FIG. 7, a region ofinterest (ROI) or a position of interest)711 is depicted in theultrasound image for a doctor to treat or examine an affected area of apatient.

Referring to 720 of FIG. 7, the medical imaging apparatus 400 may matcha CT image corresponding to an inspiratory state with the ultrasoundimage corresponding to the expiratory state and displays a matchedimage. A structural or spatial position of the object may vary dependingon a respiratory state. As shown in 720 of FIG. 7, a position of an ROI721 in the CT image corresponding to the inspiratory state does notcoincide with a position of the ROI 711 in the ultrasound imagecorresponding to the expiratory state.

Furthermore, referring to 730 of FIG. 7, the medical imaging apparatus400 may match a CT image corresponding to an expiratory state with theultrasound image corresponding to the expiratory state and display amatched image. As shown in 730 of FIG. 7, a position of an ROI 731 inthe CT image corresponding to the expiratory state coincides with theposition of the ROI 711 in the ultrasound image corresponding to theexpiratory state.

Thus, the medical imaging apparatus 400 may match a CT image and anultrasound image corresponding to the same respiratory state and displaya matched image. In this case, the respiratory state may be classifiedinto two states, i.e., expiratory and inspiratory states. Furthermore,an inspiratory state may be divided into first through N-th inspiratorystates depending on the degree to which the object inhales. Similarly,an expiratory state may be classified into first through N-th expiratorystates according to the degree to which the object exhales.

FIGS. 8A and 8B are diagrams for explaining an example where images offirst and second types are displayed on the medical imaging apparatus400, according to an exemplary embodiment.

The medical imaging apparatus 400 may display at least one of images ofa first type and at least one of images of a second type.

Referring to FIG. 8A, the medical imaging apparatus 400 may display afirst ultrasound image 811 of a second type and first through third CTimages 821 through 823 of a first type. The first ultrasound image 811of the second type and the first through third CT images 821 through 823of the first type may be images depicting the same object and maycorrespond to different respiratory states of the object.

As shown in FIG. 8A, the medical imaging apparatus 400 may display thefirst ultrasound image 811 corresponding to a first respiratory stateand the first through third CT images 821 through 823 corresponding tothe first respiratory state. Furthermore, the medical imaging apparatus400 may display the first through third CT images 821 through 823 insuch a manner that they overlap one another or are separately arranged.

Referring to FIG. 8B, the medical imaging apparatus 400 may displayfirst through third ultrasound images 811 through 813 of a second typeand first through third CT images 821 through 823 of a first type.

As shown in FIG. 8B, the medical imaging apparatus 400 may display thefirst through third ultrasound images 811 through 813 corresponding to afirst respiratory state and the first through third CT images 821through 823 corresponding to the first respiratory state. Furthermore,the medical imaging apparatus 400 may display the first through thirdultrasound images 811 through 813 in such a manner that they overlap oneanother or are separately arranged. Similarly, the medical imagingapparatus 400 may display the first through third CT images 821 through823 in the overlapping or separate manner.

FIGS. 9A through 9C are diagrams illustrating results of performingregistration on images selected by a user, according to an exemplaryembodiment.

FIG. 9A illustrates selection of at least two of a plurality of imagesdisplayed on the medical imaging apparatus 400. As shown in FIG. 9A, themedical imaging apparatus 400 may display images 811 through 813 of asecond type and images 821 through 823 of a first type. A user mayselect one from among the displayed images 821 through 823 of the firsttype as well as one from among the displayed images 811 through 813 ofthe second type.

As shown in FIG. 9A, when a screen of the medical imaging apparatus 400is formed as a touch screen, the user uses a hand to select images ofthe first and second types. The medical imaging apparatus 400 mayreceive touch inputs 901 and 902 from the user and match images of thefirst and second types based on the received touch inputs 901 and 902.

Furthermore, although FIG. 9A shows that the medical imaging apparatus400 receives a user input as a touch input, the medical imagingapparatus 400 may receive a user input via a user interface. In thiscase, the user interface may be a touch panel for sensing a touch input,and the touch panel may be integrated with the display 430) of themedical imaging apparatus 400.

Referring to FIGS. 9B and 9C, the medical imaging apparatus 400 mayreceive the touch inputs 901 and 902 as a user input and match one ofthe images of the first type with one of the images of the second type.The medical imaging apparatus 400 may display a matched image andresultant information of the matched image. The resultant information ofthe matched image may include pieces of information about whetherrespiratory states for two images coincide with each other, the numberof main points in the two images that coincide with each other fromamong all main points of the object, and precision in registrationbetween the two images. Furthermore, the medical imaging apparatus 400may display precision by using an error in a position of onecorresponding point between two images. The medical imaging apparatus400 may determine whether the two images are acquired from the object inthe same or different respiratory states, based on the error in aposition of one corresponding point. One of ordinary skill in the artwill understand that the resultant information of the matched image mayinclude pieces of information other than those described above.

As shown in FIG. 9B, the medical imaging apparatus 400 may match a firstimage of a first type with a second image of a second type. In detail,the first image of the first type may be a CT image corresponding to aninspiratory state of the object, and the second image of the second typemay be an ultrasound image corresponding to an expiratory state of theobject. Since respiratory states for the first image of the first typeand the second image of the second type are different from each other,positions of corresponding main points of the object in the first andsecond images may be different from each other. Thus, when the first andsecond images are matched together, precision in registration may belower than that when images corresponding to the same respiratory stateare matched. The medical imaging apparatus 400 may display“inconsistency in respiratory state”, “coincidence of 48 points fromamong 100 points”, and “precision of 48%” between the first and secondimages as the resultant information of a matched image.

As shown in FIG. 9C, the medical imaging apparatus 400 may match asecond image of a first type with a second image of a second type. Indetail, the second image of the first type may be a CT imagecorresponding to an expiratory state of an object, and the second imageof the second type may be an ultrasound image corresponding to theexpiratory state of the object. Since respiratory states for the secondimages of the first and second types are the same as each other,corresponding main points of the object in the second images of thefirst and second types may be located at the same position. Thus, whenthe second images of the first and second types are matched together,precision in registration may be higher than that when imagescorresponding to different respiratory states are matched. The medicalimaging apparatus 400 may display “consistency in respiratory state”,“coincidence of 99 points from among the entire 100 points”, and“precision of 99%” between the second images of the first and secondtypes as resultant information of a matched image.

The medical imaging) apparatuses described above may be implementedusing hardware components, software components, or a combinationthereof. For example, the apparatuses and components illustrated in theexemplary embodiments may be implemented using one or moregeneral-purpose or special-purpose computers, such as a processor, acontroller, an arithmetic logic unit (ALU), a digital signal processor,a microcomputer, a field programmable array (FPA), a programmable logicunit (PLU), a microprocessor or any other device capable of respondingto and executing instructions in a defined manner.

A processing device may run an operating system (OS) and one or moresoftware applications running on the OS. The processing device also mayaccess, store, manipulate, process, and create data in response toexecution of software.

For convenience, although a single processing device may be illustratedfor convenience, one of ordinary skill in the art will appreciate that aprocessing device may include a plurality of processing elements and/ora plurality of types of processing elements. For example, a processingdevice may include a plurality of processors or a processor and acontroller. In addition, the processing device may have differentprocessing configurations such as parallel processors.

Software may include a computer program, a piece of code, aninstruction, or one or more combinations thereof and independently orcollectively instruct or configure the processing device to operate asdesired.

Software and/or data may be embodied permanently or temporarily in anytype of machine, component, physical equipment, virtual equipment,computer storage medium or device, or in a transmitted signal wave so asto be interpreted by the processing device or to provide instructions ordata to the processing device. The software also may be distributed overnetwork-coupled computer systems so that the software is stored andexecuted in a distributed fashion. In particular, the software and datamay be stored in one or more computer-readable recording media.

The methods according to the exemplary embodiments may be recorded innon-transitory computer-readable recording media including programinstructions to implement various operations embodied by a computer. Thenon-transitory computer-readable recording media may also include, aloneor in combination with the program instructions, data files, datastructures, and the like. The program instructions recorded in thenon-transitory computer-readable recording media may be designed andconfigured specially for the exemplary embodiments or be known andavailable to those of ordinary skill in computer software.

Examples of non-transitory computer-readable recording media includemagnetic media such as hard disks, floppy disks, and magnetic tape;optical media such as CD ROM discs and DVDs; magneto-optical media suchas optical discs; and hardware devices that are specially configured tostore and perform program instructions, such as ROM, RAM, flash memory,and the like.

Examples of program instructions include both machine code, such as thatproduced by a compiler, and higher level code that may be executed bythe computer using an interpreter.

The above-described hardware devices may be configured to act as one ormore software modules in order to perform the operations of theabove-described embodiments, or vice versa.

While one or more exemplary embodiments have been described withreference to the figures, it will be understood by those of ordinaryskill in the art that various modifications and changes in form anddetails may be made from the above descriptions without departing fromthe spirit and scope as defined by the following claims. For example,adequate effects may be achieved even if the above techniques areperformed in a different order than described above, and/or theaforementioned elements, such as systems, structures, devices, orcircuits, are combined or coupled in different forms and modes than asdescribed above or are replaced or supplemented by other components ortheir equivalents.

Thus, the scope of the present inventive concept is defined not by thedetailed description thereof but by the appended claims and theirequivalents.

What is claimed is:
 1. A method of operating a medical imagingapparatus, the method comprising: acquiring a first image of a firsttype corresponding to a first respiratory state of an object; acquiringa first ultrasound image corresponding to the first respiratory state ofthe object and a second ultrasound image corresponding to a secondrespiratory state of the object; generating a second image of the firsttype in which the second respiratory state of the object is predicted,based on anatomical information extracted from the first ultrasoundimage and the second ultrasound image; and registering the second imageof the first type and the second ultrasound image.
 2. The method ofclaim 1, wherein the generating the second image of the first typecomprises predicting a second anatomical information corresponding tothe second respiratory state of the object from a first anatomicalinformation corresponding to the first respiratory state of the objectin the first image of a first type, based on the anatomical informationextracted from the first ultrasound image and the second ultrasoundimage; and generating the second image of the first type based on thesecond anatomical information.
 3. The method of claim 2, wherein thepredicting the second anatomical information comprises determiningmotion information of the object with respect to a respiratory state,based on the anatomical information extracted from the first ultrasoundimage and the second ultrasound image; predicting the second anatomicalinformation corresponding to the second respiratory state of the objectby applying the motion information to the first image of the first type.4. The method of claim 3, wherein the determining the motion informationof the object comprises determining a spatial transformation between thefirst ultrasound image and the second ultrasound image, based on theanatomical information extracted from the first ultrasound image and thesecond ultrasound image; and determining the motion information of theobject with respect to the respiratory state, based on at least oneparameter describing the spatial transformation between the firstultrasound image and the second ultrasound image.
 5. The method of claim2, wherein the generating the second image of the first type comprisestransforming the first image of the first type to generate the secondimage of the first type, based on the first anatomical information andthe second anatomical information.
 6. The method of claim 1, furthercomprising displaying a registered image.
 7. The method of claim 1,further comprising displaying at least one of the first and secondimages of the first type and at least one of the first ultrasound imageand the second ultrasound image.
 8. The method of claim 7, wherein thedisplaying at least one of the first and second images of the first typeand at least one of the first ultrasound image and the second ultrasoundimage comprises displaying at least two of the first and second imagesof the first type and the first and second ultrasound images in such amanner that the at least two images overlap each other.
 9. The method ofclaim 7, further comprising receiving a user input for selecting the atleast one of the first and second images of the first type and the atleast one of the first and second ultrasound images; and registering theselected at least one image of the first type with the selected at leastone image of the first and second ultrasound images.
 10. The method ofclaim 3, wherein the determining the motion information of the objectcomprises determining the at least one parameter for acquiring motioninformation indicating a spatial transformation between the first andsecond ultrasound images; determining a value of the at least oneparameter based on the spatial transformation therebetween; anddetermining the motion information based on the determined value of theat least one parameter.
 11. The method of claim 10, wherein the spatialtransformation is based on at least one of a position, rotation, and asize of the object.
 12. The method of claim 3, wherein the determiningthe motion information of the object comprises acquiring positioninformation of the object and determining the motion information fromthe first and second ultrasound images respectively acquired in thefirst respiratory state and second respiratory state corresponding tothe acquired position information.
 13. The method of claim 1, whereinthe first image of the first type is one of an optical coherencetomography (OCT) image, a CT image, a magnetic resonance (MR) image, anX-ray image, a single photon emission computed tomography (SPECT) image,a positron emission tomography (PET) image, a C-arm image, a PET-CTimage, a PET-MR image, and a fluoroscopy image.
 14. The method of claim1, wherein the first respiratory state is a inspiratory state of theobject, wherein the second respiratory state is a expiratory state ofthe object.
 15. The method of claim 1, wherein the first and the secondimages of the first type and the first and second ultrasound images are2D or 3D images.
 16. A medical imaging apparatus comprising: an imageprocessor configured to acquire a first image of a first typecorresponding to a first respiratory state of an object, and acquire afirst ultrasound image corresponding to the first respiratory state ofthe object and a second ultrasound image corresponding to a secondrespiratory state of the object; a controller configured to generate asecond image of the first type in which the second respiratory state ofthe object is predicted, based on anatomical information extracted fromthe first ultrasound image and the second ultrasound image, and registerthe second image of the first type and the second ultrasound image. 17.A non-transitory computer-readable recording medium having recordedthereon a program for performing a method of operating a medical imagingapparatus, wherein the method comprises: acquiring a first image of afirst type corresponding to a first respiratory state of an object;acquiring a first ultrasound image corresponding to the firstrespiratory state of the object and a second ultrasound imagecorresponding to a second respiratory state of the object; generating asecond image of the first type in which the second respiratory state ofthe object is predicted, based on anatomical information extracted fromthe first ultrasound image and the second ultrasound image; andregistering the second image of the first type and the second ultrasoundimage.
 18. A method of operating a medical imaging apparatus, the methodcomprising: acquiring a first image of a first type corresponding to ainspiratory state of an object; acquiring a first image of a second typecorresponding to the inspiratory state of the object and a second imageof the second type corresponding to a expiratory state of the object;generating a second image of the first type in which the expiratorystate of the object is predicted, based on anatomical informationextracted from the first image of the second type and the second imageof the second type; and registering the second image of the first typeand the second image of the second type.